1. Field of the Invention
The present invention relates to a device, a method, and a system for non-linear editing of video, and to a computer-readable recording medium on which a program for video editing is recorded.
2. Description of the Related Art
Conventionally, video editing work has been performed by using an editing device called a non-linear editor. The non-linear editor can copy video and/or audio data (called “material” below) which has been recorded on an original tape onto a random-access recordable and readable disk-type recording medium (simply called a “disk” below), such as a hard disk, once, and can then read out a plurality of materials recorded on the disk in a user-defined time interval. Then, it can perform a variety of image special effects (called “effects” below) on these materials for output on a monitor or a tape. According to this device, quick access can be made to each editing point (cut), which reduces the time required for the editing work compared to linear editing in which editing is performed directly on the tape.
In the non-linear editor, important factors for performance evaluation are how fast a series of the above-described editing operations can be performed than that for real-time processing, or among types of effect processing that are complicated and cannot be performed in real time, how rapidly rendering processing can be performed. Here, rendering processing refers to processing that converts image signals for creating an image to be displayed and whose result is written on a recording medium.
In the conventional non-linear editor, it is common for internal processing to use a processing method called streaming processing or a processing method called frame-by frame processing.
The streaming processing method is a form of processing used in an editor mainly utilizing hardware. It buffers material retrieved from a disk asynchronously and a plurality of types of pipe-lined effect processing is performed thereon mutually and simultaneously. It should be noted that the asynchronous processing refers to processing for reading data in advance before outputting the data, while synchronous processing refers to processing performed by following time (that is, synchronously) along the flow (movement) of output data. In “pipelined” types of processing, individual types of processing overlap in time at each stage in a plurality of functional units (stages) formed by coupling stages in series so that predetermined types of processing are performed, respectively. That is, a so-called multi-task (parallel) method is applied in order to attempt to reduce the overall processing time.
FIG. 17 shows an overview of the streaming processing method while FIG. 18 shows a timing relationship when each video frame is processed at each stage. The term “video frame” refers to one static image that is a basic unit forming image data. It is simply called a “frame” below. In FIG. 18, (A) shows a time axis, where the length of one frame T in a normal video playback condition is a basic unit, (B) shows the output from a disk 201, (C) shows the output from a buffer 202, (D) shows the output form an effect module 203, and (E) shows the output from an effect module 204. In (B) through (E), one frame is shown in each of F1 to F5. Here, it is assumed that the operating speed of the disk 201 is slower than those of the other resources (effect modules 203 and 204).
In the streaming processing method, as shown in FIG. 17 and FIG. 18, parts (B) and (C), material pre-read from the disk 201 for several frames is stored in the buffer 202, and then the frames are output sequentially in realtime. Further, as shown in FIG. 17 and FIG. 18, parts (D) and (E), each type of corresponding effect processing is performed by the effect modules 203 and 204 on material (each frame) sequentially output from the buffer 202. Here, neither type of effect processing performed by the effect modules 203 and 204 is complicated. Rather, if those types of processing are simple (light) enough for maintaining the real-time characteristic, effect processes which are synchronous with output data 205 to an external device (such as monitor and tape) are possible.
On the other hand, the frame-by-frame processing method is used in an editor mainly utilizing software and is a one-frame-completed type processing method.
FIG. 19 shows an overview of the frame-by-frame processing method while FIG. 20 shows a timing relationship when each video frame is processed at each step in the method. In FIG. 20, (A) shows the time axis, where one frame length in a normal video playback condition is a basic unit, (B) shows the output from a disk 211, (C) shows the output from an effect module 213, (D) shows the output from an effect module 214, and (E) shows the output from a memory 212. In (B) through (E), one frame is shown in each of F1 and F2. Here, it is assumed that the operating speed of the disk 211 is slower than those of the other resources (effect modules 213 and 214).
In the frame-by-frame processing method, as shown in FIGS. 19 and 20, a series of processing on one frame is completed before starting a series of processing performed on the next frame. More specifically, as shown in (B) to (E) in FIG. 20, each type of effect processing is performed by the effect modules 213 and 214 on a frame F1 read out from the disk 211. Once the final result (processed frame) is written in the memory 212, the processed frame) is output from the memory 212. Then, a next frame F2 is read out from the disk 211, and each types of effect processing is performed on the frame F2 by the effect modules 213 and 214 in the same manner as above. Once the processed frame, that is the final result, is written in the memory 212, the processed frame is output from the memory 212. Thereafter, similar types of processing follow.
The streaming processing utilizes hardware, which permits rapid processing. Particularly, as long as the steps in the effect processing are not very complicated, processing close to real-time processing is possible. For example, in FIG. 18, the time required for processing one frame is τ1/5, which allows the reduction of processing time where a time required for all types of processing on five frames F1 to F5 is τ1.
On the other hand, the streaming processing has the following problems:    (a) Since it is performed by hardware, a change cannot be handled in a flexible manner when the video signal format, such as size and/or aspect ratio (the ratio of length to width) of an image, is changed. In other words, different hardware must be prepared for each video signal format in order to handle a plurality of different video signal formats;    (b) Since a hardware module performing each effect is essentially fixed, it is not possible to use a part of the processing in order to perform another type of effect processing or to exchange the order of a plurality of types of processing when one effect includes the plurality of types of processing; and    (c) In general, the streaming processing method is not compatible with rendering by software. The reason is as follows: In general, hardware processing causes a delay during a period from the time when a type of processing is requested to the time when the processing is actually performed while the period for which the processing is actually performed is significantly shorter than that of software processing. On the other hand, while the software processing is flexible and has a higher precision, the processing time is longer than that of hardware processing. Therefore, in the streaming processing, in order to permit the mixture of hardware processing and software processing, it is necessary to perform several forms of hardware processing continuously for higher efficiency. That is, in a system using the streaming processing method, it is necessary, in consideration of both hardware processing and software processing, to prepare routines (paths) which are suitable for each type of processing. Thus, it is not easy to incorporate rendering software, for processing or creating an image, in an editor using the streaming processing method.
On the other hand, in the frame-by-frame processing, as described above, since it is implemented mainly by using software, unlike the streaming processing method, it is possible to handle a change in the video signal format or a change in the order of processing steps within one effect in a flexible manner.
However, the frame-by-frame processing, as shown in FIG. 20, is the one-frame completion type method according to a single task method in which several types of processing are performed on one frame sequentially from a previous stage (effect module) to a subsequent stage. Thus, the required time τ2 for processing one-frame is obtained by adding a time t1 required for reading out material from a disk, times t2 and t3 required for several types of effect processing, and a time t4 required for outputting from a memory. Therefore, in the frame-by-frame processing method, the speed of rendering processing is extremely low, which makes real-time processing difficult.